CS 224 N : Natural Language Processing

The objective of this project is to analyze the performance of a class-based language model and compare it to the performance of traditional n-gram language models. Class-based language models are well-studied, as is the use of clustering to learn classes of words. However, it seems fairly standard across the literature to use hard-clustering i.e. assign each word to a single class and then to use these classes in a class n-gram language model. Also, word clustering seems to be often done in conjunction with document clustering, allowing document class to be used in determining word class and vice versa. We hoped to do something a bit different from what appeared to be the standard approach. We, by no means, think our approach is novel, but we found relatively little literature whose work we seemed to be repeating. First and foremost, we clustered words entirely on part of speech information. Secondly, we believe that a single word’s class can vary according to its context, suggesting a soft-membership clustering approach that would give a probability distribution over classes given a word and its context. Finally, since word class is a hidden variable, with the actual text representing the observables , we have attempted to construct HMM-based language models. Ultimately, we found our learned hidden state class-based language model performed significantly worse than traditional n-gram word language models. We have ideas about how our approach might be improved with future research, but we are not convinced that such work is really warranted.